Added support for Databricks Apps in DABs #1928

andrewnester · 2024-11-21T16:26:28Z

Changes

Now it's possible to configure new app resource in bundle and point it to the custom source_code_path location where Databricks App code is defined.

On databricks bundle deploy DABs will create an app. All consecutive databricks bundle deploy execution will update an existing app if there are any updated

On databricks bundle run <my_app> DABs will execute app deployment. If the app is not started yet, it will start the app first.

Bundle configuration

bundle:
  name: apps

variables:
  my_job_id:
    description: "ID of job to run app"
    lookup:
      job: "My Job"
  databricks_name:
    description: "Name for app user"
  additional_flags:
    description: "Additional flags to run command app"
    default: ""
  my_app_config:
    type: complex
    description: "Configuration for my Databricks App"
    default:
      command:
        - flask
        - --app
        - hello
        - run
        - ${var.additional_flags}
      env:
        - name: DATABRICKS_NAME
          value: ${var.databricks_name}

resources:
  apps:
    my_app:
      name: "anester-app" # required and has to be unique
      description: "My App"
      source_code_path: ./app # required and points to location of app code
      config: ${var.my_app_config}
      resources:
        - name: "my-job"
          description: "A job for app to be able to run"
          job:
            id: ${var.my_job_id}
            permission: "CAN_MANAGE_RUN"
      permissions:
        - user_name: "[email protected]"
          level: "CAN_VIEW"
        - service_principal_name: "my_sp"
          level: "CAN_MANAGE"

targets:
  dev:
    variables:
      databricks_name: "Andrew (from dev)"
      additional_flags: --debug
  
  prod:
    variables:
      databricks_name: "Andrew (from prod)"

Execution

databricks bundle deploy -t dev
databricks bundle run my_app -t dev

If app is started

✓ Getting the status of the app my-app
✓ App is in RUNNING state
✓ Preparing source code for new app deployment.
✓ Deployment is pending
✓ Starting app with command: flask --app hello run --debug
✓ App started successfully
You can access the app at <app-url>

If app is not started

✓ Getting the status of the app my-app
✓ App is in UNAVAILABLE state
✓ Starting the app my-app
✓ App is starting...
....
✓ App is starting...
✓ App is started!
✓ Preparing source code for new app deployment.
✓ Downloading source code from /Workspace/Users/...
✓ Starting app with command: flask --app hello run --debug
✓ App started successfully
You can access the app at <app-url>

Tests

Added unit and config tests + manual test.

--- PASS: TestAccDeployBundleWithApp (404.59s)
PASS
coverage: 36.8% of statements in ./...
ok      github.com/databricks/cli/internal/bundle       405.035s        coverage: 36.8% of statements in ./...

bundle/internal/tf/schema/resource_app.go

## Changes Added support for bundle generate and bind for Apps ## Tests - [ ] Add E2E test

eng-dev-ecosystem-bot · 2024-12-18T13:34:59Z

Test Details: go/deco-tests/12394072048

bundle/apps/upload_config.go

pietern · 2024-12-20T14:30:02Z

bundle/apps/interpolate_variables_test.go

+	diags := bundle.Apply(context.Background(), b, InterpolateVariables())
+	require.Empty(t, diags)
+	require.Equal(t, []any([]any{map[string]any{"name": "JOB_ID", "value": "123"}}), b.Config.Resources.Apps["my_app_1"].Config["env"])
+	require.Nil(t, b.Config.Resources.Apps["my_app_2"].Config)


This test works but doesn't actually use the translation map.

I suspect you intend to make sure that the conversion of bundle-internal field references to Terraform resource field references is undone so that we can interpolate the IDs ourselves. But the test runs against a bundle-internal field reference.

Is there an alternative approach such that we don't have to convert back-and-forth? Maybe we completely skip this path during variable resolution? Or only skip resources.* references under resources.apps.*.config.**? Then we have less coupling with the TF resource interpolation.

I prefer to keep conversion to TF for all resources and then having this separate mutator which converts it back. The reasons are:

We will still need to keep InterpolateVariables mutator in both cases (with or without the conversion)

If we decide to add more fields to be processed this way we will need to add 2 changes in 2 places: a conditional skip in TF mutator, and the processing of the field in InterpolateVariables mutator. Without the skip we just need to add the change in 1 place (InterpolateVariables)

That works for me.

The test now uses the TF notation, so it will fail when that is changed.

bundle/apps/upload_config.go

bundle/apps/validate.go

bundle/config/resources/apps.go

bundle/deploy/terraform/util.go

bundle/internal/schema/annotations.yml

integration/bundle/apps_test.go

bundle/apps/interpolate_variables.go

pietern · 2024-12-27T15:42:20Z

bundle/apps/interpolate_variables_test.go

+	diags := bundle.Apply(context.Background(), b, InterpolateVariables())
+	require.Empty(t, diags)
+	require.Equal(t, []any([]any{map[string]any{"name": "JOB_ID", "value": "123"}}), b.Config.Resources.Apps["my_app_1"].Config["env"])
+	require.Nil(t, b.Config.Resources.Apps["my_app_2"].Config)


That works for me.

The test now uses the TF notation, so it will fail when that is changed.

pietern · 2024-12-27T15:49:43Z

bundle/apps/upload_config.go

+		// It means we need to write app.yml file with the content of the config field
+		// to the remote source code path of the app.
+		if app.Config != nil {
+			appPath := strings.TrimPrefix(app.SourceCodePath, b.Config.Workspace.FilePath)


@fjakobs @ilyakuz-db FYI, this is a potential problem for source-linked deployments. We assume here that files are deployed under ${workspace.file_path} and then write a file into it. If we were to make this work for source-linked deployments, this would have to write a file to the source folder itself, which in turn means we'll get validation errors (we check that the user does not have both an app.yml and a configuration section in the bundle configuration for their app.

The fact that we have to think about this (and potentially work around it) illustrates the problem that this duality brings.

bundle/apps/upload_config.go

bundle/apps/upload_config_test.go

bundle/apps/validate.go

bundle/config/mutator/run_as_test.go

bundle/deploy/terraform/util.go

github-actions · 2024-12-31T14:17:22Z

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

PR number: 1928
Commit SHA: 7693f0add0ab88bd49c109047cedc75cc16edd42

Checks will be approved automatically on success.

andrewnester added the do-not-merge label Nov 21, 2024

andrewnester requested review from pietern, denik and shreyas-goenka November 21, 2024 16:26

andrewnester temporarily deployed to test-trigger-is November 21, 2024 16:26 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 21, 2024 16:33 — with GitHub Actions Inactive

andrewnester commented Nov 21, 2024

View reviewed changes

bundle/internal/tf/schema/resource_app.go Outdated Show resolved Hide resolved

andrewnester temporarily deployed to test-trigger-is November 21, 2024 16:36 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 21, 2024 16:46 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 26, 2024 15:14 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 26, 2024 15:15 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 26, 2024 15:17 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 27, 2024 14:11 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 28, 2024 14:33 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 28, 2024 15:39 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 28, 2024 15:40 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 28, 2024 15:41 — with GitHub Actions Inactive

andrewnester force-pushed the feature/apps branch from bc4c9ac to d72b03e Compare November 29, 2024 11:51

andrewnester temporarily deployed to test-trigger-is November 29, 2024 11:51 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is December 2, 2024 15:05 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is December 18, 2024 11:34 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is December 18, 2024 11:35 — with GitHub Actions Inactive

Added support for bundle generate and bind for Apps (#1946)

e622ab5

## Changes Added support for bundle generate and bind for Apps ## Tests - [ ] Add E2E test

andrewnester temporarily deployed to test-trigger-is December 18, 2024 12:16 — with GitHub Actions Inactive

fixed test for gcp

db516b6

andrewnester temporarily deployed to test-trigger-is December 18, 2024 13:34 — with GitHub Actions Inactive

denik reviewed Dec 19, 2024

View reviewed changes

bundle/apps/upload_config.go Show resolved Hide resolved

bundle/apps/upload_config.go Outdated Show resolved Hide resolved

bundle/apps/upload_config.go Show resolved Hide resolved

Added same source validation and made thread safe

2f8df81

andrewnester temporarily deployed to test-trigger-is December 19, 2024 13:00 — with GitHub Actions Inactive

andrewnester requested a review from denik December 20, 2024 11:58

pietern reviewed Dec 20, 2024

View reviewed changes

andrewnester added 2 commits December 23, 2024 15:07

Merge remote-tracking branch 'origin/main' into feature/apps

0583ded

feedback fixes

86630b6

andrewnester temporarily deployed to test-trigger-is December 23, 2024 15:12 — with GitHub Actions Inactive

andrewnester requested review from ilyakuz-db and pietern December 24, 2024 12:09

pietern reviewed Dec 27, 2024

View reviewed changes

fixs

4b4cc42

andrewnester temporarily deployed to test-trigger-is December 30, 2024 11:44 — with GitHub Actions Inactive

andrewnester requested a review from pietern December 30, 2024 11:44

always set description field

7693f0a

andrewnester temporarily deployed to test-trigger-is December 31, 2024 14:17 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for Databricks Apps in DABs #1928

Added support for Databricks Apps in DABs #1928

andrewnester commented Nov 21, 2024 •

edited

Loading

eng-dev-ecosystem-bot commented Dec 18, 2024

pietern Dec 20, 2024

andrewnester Dec 23, 2024

pietern Dec 27, 2024

pietern Dec 27, 2024

pietern Dec 27, 2024

github-actions bot commented Dec 31, 2024

Added support for Databricks Apps in DABs #1928

Are you sure you want to change the base?

Added support for Databricks Apps in DABs #1928

Conversation

andrewnester commented Nov 21, 2024 • edited Loading

Changes

Bundle configuration

Execution

Tests

eng-dev-ecosystem-bot commented Dec 18, 2024

pietern Dec 20, 2024

Choose a reason for hiding this comment

andrewnester Dec 23, 2024

Choose a reason for hiding this comment

pietern Dec 27, 2024

Choose a reason for hiding this comment

pietern Dec 27, 2024

Choose a reason for hiding this comment

pietern Dec 27, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 31, 2024

andrewnester commented Nov 21, 2024 •

edited

Loading